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ABSTRACT ' . 

♦ 

Weaknesses in evaluations often can. be traced to structural 
limitations in the positions of evaluation researchers. Conventional 
human relations techniques often are an insufficient basis for 
securing strong support for evaluation research. Strategies for 
increasing evaluation research leverage are reviewed. Alignment- of ( 
evaluation research with regulatory bodies with, authority to suspend 
public program expenditures is advocated. Several likely obstacles 
in the development of the regulatory evaluation model are anticipated 
and addressed. 
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LEVERAGE AND EVALUATION EFFECTIVENESS 

• 

Mere presence in a social program domain doe's not assure 
' evaluation researchers of the support they require to "contribute 
effectively. Reluctance of program operators to specify objectives, . 
to agree to random assignment of potential elients to control groups, 
J to permit systematic observation of service transactions, to allow 
thorough, independent pre- and postmeasurement of service recipients 
on dependent variables, to attend seriously^pfto the implications of 
evaluation findings, and to ^ssent to publication of disappointing 
findings are among the objrtacles which evaluation researchers 
frequently encouter. A some instances, of course, evaluators enjoy 
very strong suppor^and face few if any of the problems listed above. 
In other instances evaluators experience all of these obstacles and mor*. 

Experienced evaluation researchers are accustomed to dealing with 
adversit/T They make judicious guesses about program intentions and \ 
lear/to live with criticism for addressing the wrong questions. They 

jlarly us* quasi-experimental and even pre-experimental designs, 
address process variables" which are poor substitutes for measures of j 

outcome variables, and make use of flawed data available through service j 

^* *^ i 

. ■ 

records. In making these ^methodological compromises, evaluators ; 
ultimately invite critical comments from their colleagues (see, for 
example, Bernstein S Freeman, 1975 and Cook 5 Gruder, 1978) . 

At issue, of course, is not simply the evaluator's interest in 
carrying out sound research but the public interest in effective social 
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outcome variables, and make use of flawed data available through service 

■■ "\ 

records. In making these ^methodological compromises, evaluators 

• \ 

ultimately invite critical comments from their colleagues (see, for 
example, Bernstein 3 Freeman, 1975 and Cook 5 Gruder, 1978) . 

At issue, of course, is not simply the evaluator's interest in 
carrying out sound research but the public interest in effective social ' 

/■ 

/ • 

// 



ERIC 



- 2 - 



\ 



ERIC 



programs. 'Assuming^ worjpt that social^programs' are hajrtnless, the 

public has reason to be concerned about the taxy i^pli^tions of 

r m 

publicly supported programs which continue to be justified on faith 
rather than evidence . Of potential concern are not only programs 
receiving direct public funds but privately financed endeavors 
dependent on their tax exempt status. While federally funded demonstration 
programs now regularly receive evaluation attention, many ongping programs 
continue to receive no evaluation attention at all. The public interested 
in efficient use of? resources, therefore, has reason to be concerned not 
only about evaluations which are equivocal because of their soft 
methodology but about the programs entirely untouched by evaluation 
research. 

What can be done to extend the evaluation research dqjna^n and 
increase its leverage? Evaluators are accustomed to finding themselves 
the advocates not only for powerful methodology but also for evaluation 
itself. As advocates they typically try to educate and persuade. They-, 
rely extensively on human relations techniques to establish vtheir 
importance, to sustain tjhe interest and cooperation of program personnel, 
and to persuade clients to attend devaluation results {see J for example, 
Caro, 1977). 

Clearly some evaluators are highly su^essful in making a case for 
evaluation and in persuading fujnding agencies and program operators to 
provide the support iifcessary for powwful evaluations. In estimating 
what evaluators can expect to accomplish through education, advocacy, 
* and human relations, it is useful, hoxever, to examine the perspectives 
of other key actors in the social program domain. Because of the 
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-importance "of finances and control over operations, funding agencies 
and administrators play particularly 'Important roles. in determining 
the fate of evaluation concerns.. In principle, both those who allocate 
resources and those who -administer programs have reason to support 
program evaluation.. They should be committed to efficient and effective 
use of scarce resources. They should appreciate the contribution which 

•evaluation can make to'progTam development. The organizational literature 
and. the experience of evaluators, however, suggest that other forces may 
dampen the enthusiasm of funding agencies anXadministrators. for 

evaluation. > 

Because administrators have received more attention in the 
literature, it is convenient to consider their perspective on evaluation 

• * * 

first. Some years ago Etzioni, <1960) made the useful observation that 
organizations are not simply concerned with realization of program 
objectives but such other matters as organizational survival. Public 
objectives are sometimes less important than unpublicized organizational 
goals. Administrators do make commitments to programs for reasons which 
are not/made public, e.g., their own career ambitions, cultivation of . 
important external support, and loyalty to stiff. Evalutions addressed 
€o official objectives in some instances might not only embarrass the 
organization byy showing modest res^ts^ut invite unwelcome questions 

about unofficial reasons for commftment to the program. As Schulberg 5 
Baker (1968) point out, administrators for theses-reasons often are careful 
in identifying programs for which they invite evaluation attention. » 
Although contraction in public funding for social programs is 
\ <fcited commonly as an argument for expanded emphasis on program evaluation, 
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scarce resources can'. contribute to the forces which undermine evaluation. 
Publicly funded social programs typically are addressed to tough 
residual problems.* To secure resources in a highly competitive market, 
administrators have learned that it can be useful to project a « 
"miracle worker'-' image. They learn to convey the conviction 'that with 
modest resources they will achieve -dramatic results. As Campbell (1969) 
puts lit "Specific reforms are advocated as though they were certain to 
be successful." Skillful operators have learned how Ho use evidence of 
early, apparently promising results to proclaim success and to command 
additional resources'. Sometimes these administrators welcome the 
presence of evaluation researchers as a means of enhancing their 

♦ 

credibility and prestige. The evaluatprs, of course, are welcome only 
to conduct studies which do not challenge fundamental program premises. 
In Campbell's terms, these administrators are "trapped" by their 
exaggerated commitments and cannot afford an honest evaluation: 

.Similarly, funding agencies whether public or private have reasqns 
for ambivalence about evaluation. In issuing grants and contracts, 
funding bodies are well advised to scrutinize applicants carefully. 
In principle, funding agencies should tseek evidence of effectiveness 
in program performance as a guide to continuing funding decisions . 
Funding agencies, however, may be trapped in much the same way as 
^administrators. Seeking to maximize what they can accomplish with 
scarce resources, funding bodies are attracted to those who promise to ^ 
accomplish a great deal at a modest cost. Program sponsors, therefore, 
are highly -vulnerable to being victimized by -over-advocacy . Sponsors 
'may be\ble to afford honest evaluations exposing serious limitation 

» 



in a few of their programs, but evidence of pervasive weaknesses in 
supported pro gr-ams would erode their own credibility. To protect fhe 
public confidence they enjoy, funding bodies have reason to be careful . 
about the evaluations they encourage. To the extent to which funding 
bodies attempt to accomplish a great deal with meager resources, they 
are increasingly vulnerable to being embarrassed by thorough ■ 

evaluations. " • 

An (additional force which inhibits efforts to establish a 
-constituency for evaluation research might be described as a longing 
for faith. Not only among program sponsors and program personnel but ' 
among clients, legislative bodies, and the general public, there is 
a desire to believe that problems can be solved, that certain interventions 
work. On some matters various publics are willing and even eager to be 
skeptical. Challenges to certain fundamental assumptions, however, are 
not- fully welcome because they are unsettling. The public wants to 
believe, for example, that education is beneficial and^ that physicians 
can effectively treat illness. Evaluation research is a partyof a large 
set of cultural forces-which seek increasin'g rationalization of society. 
A great deal has been accomplished over a period of several centuries 
in "advanced societies" in gaining acceptance for challenges to 
traditional practices. Yet it is important to recognize that the quest ^ 
for faith remains alive. In some sectors evaluation efforts will 
continue to attract an indifferent or even somewhat hostile response v 
because some portion of the concerned public is not prepared to have its 
faith in a program intervention challenged. 
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Increased Leverage for Evaluation 



* N . ..*._. 

In light -of the strength of the forces cojastrainajig significant 
evaluation contributions, more than education and advocacy are needed 
if' evaluation is to become a strong presence throughout the social 
programming domain. A number of^models for providing increased 
leverage for Evaluation deserve attention. 

Watchdogs and Gadflies. Organizations with a mandate to protect the 

1 r ~ — 

y the public' interest increasingly show signs of interest in evaluation 

research. Outside of formal lines of authority, organizations like the 
League of Women Voters, Common Cause, and "Nader's' Raiders" sponsor 
inquiries into various public programs. These "wat.ch^ogs"^ use their . 
prestige, the content' of their message, and persuasion to influence 
policy. Independent investigations' of the operations of public programs . 
are a well established tradition in American social reform efforts but . 
their explicit link to evaluation research is relatively new. The watch- 
, -dog who operates out of the private sector is typically, constrained 
greatly by modest financial resources. Insight of\ the more adequate 
funding of public watchdog agencies-, their rtceat interest in evaluation 
research is particularly encouraging. ^Traditionally such units limited 
T themselves to 'financial accounting. Oif a fecfcral level, the General 

Accounting Office which was created 'to serve Congress increasingly conducts 

* 

inquiries concerned with program effectiveness. The New York City > v 

Comptroller's occasional studies of program performance indicates that 
this broader conception of public accounting responsibilities is not 
entirely limited to the federal level . (See, for example, N.Y.C. Office r . 
of the. Comptroller, 1978-jfc - 
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. For th, most part the watchdog, must be satisfied to use available 
data or conduct surveys. Outside. of and antaganistic to program 
authority structures, the watchdog usually cannot conduct true experiments. 
. (Cleverly designed experimental evaluations, however, have been conducted 
by watchdog groups to test the effectiveness of programs concerned with 
discrimination in such areas as housing and public accommodations. (See, 

for example j Wienk etal.., '1379.) 

A variation on the watchdog approach is the "gadfly" approach. 
Working alone and without official sanction, the gadfly is opportunistic 
in gatering data. * Sometimes needed information is in the &>lic domain. 
Frequently the gadfly gains entry to an .organisation by con«aUng his . 
fun agenda.' Disgruntled lower echelon staff members are of^n/xey sources 
of data.' Because he has to max. 'use. of opportunities as they present ; 
themselves, the gadfly may have to be content with qualitative ^aSa. v 

Both the watchdog and gadfly models are to be encouraged ks meats 
of calling attention to matters otherwise inaccessible to ".valuators . 
Th. weakness of both models, however, are conspicuous. Limited 
opportunities for true experimentation, uncertain access' to data, and 
uncertain influence over decision-dicing mean that additional models 
are needed to extend the .evaluatio^£in. 

Evaluati- Imperialismi . Another possibility is for .valuators to 
•s..x control ov.r program qp.rations. If the evaluation researcher^. 
' becomes the program administrator, he may be able. t» us. his authority 
to d.cid. that .valuations addr.ss.d to central issues and employing 
powerful methodologies' are to be conducted. The evaluator-program 
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administrator also/may be able to arrange a budget which assures . 
sufficient funding for'strong evaluation. This approach migh^ ^ 
termed "evaluation imperialism.:' As a strong advocate .of this ^ 
approach,. Tomatzky (1979) cites the achievements of George .Fairweather 
(1964, 1969) . , Working in the Veterans Administration, Fairweather ^ 
achieved-administrative control over both a ward for mental patients 
and a complementary community facility. Not only did' Fairweather use . 
his authority to introduce innovative programming but he conducted ^ 
randomized experiments. Tomatzky points out that Fairweather fcven 
used his leyerage.as service administrator to fhift assignments of ^ 
ward staf/ to eliminate personnel as a plausible explanation "of 
differences between experimental and control groups'. Tornatzky . 
argues that evaluators are often too quick to accept a subordinate- .. 
role. If they are enterprising and resourceful, evaluators can 
acquire the authority over programming which may be needed ifthey- 
are to be able to conduct powerful experimental evaluations. 

The imperialist model makes an important contribution in calling 
Mention to the importance of authority. The .evaluator who controls , 
' decisions about what is to be evaluated and what method^ may He ployed 
is in a much better position to conduct evaluatibns which are sub- . ( 
stantively significant and methodologically strong than* the evaluator . 
'who must rely on education and^persuasion. Further, the evaluator- - 
program administrator presumably is guided 'by an interest in using ■ p 
' evaluatiqn results jto improve program operations . ^ 

Evaluation imperialism,' however, is not without its limits as a 
aodel ^increasing evaluation research leverage. Competition for 

• * 



control ovet social program operations often is substantial. In spite 
of their vigorous efforts, evaluation researchers frequent ly^wi 11 be 
unable to obtain authority over program operations. Further, those who 
do manage to achieve control over programs will finely themselves under- . 
pressures detracting from their intentions, to conduct experimental 
evaluations. In tjijeir classic article on' researcher-practitioner 
relations, Rodman and Kolodny (1964) effectively 'argued that an inherent 
strain {iiv^des the^two roles. Those who\atterapt to combine the roJ.es 
find it difficult to reconcile the two sets of Tesponsibilities . Faced 
with aboard, practitioners, and perhaps client advocates who are'opposed 
to random assignment of clients to control groups, the program 
administrator-evaluator may find maintenance of good rejpationsr with 
key groups"ln his operational domain more important tttan true 
experimentation. Confronted by financial* constraints, he may have to 
compromise .with his intention to invest significantly in evaluation 
research. • Perhaps most importantly, .to. gain and maintain administrative 

f * 

a , 

authority and to be able to generate external financial support-, the 

evaluation imperialist is likely to be trapped by over-advocacy in mjich 
f 

the same fashion as other administrators. While evaluation imperialism; 

. * * m 

can be useful in extending the evaluation domain, it is not sufficient 
as a general mea^s of 'assuring evaluation the leverage it deserves. • 

Regulatory Evaluation . A third approach which aligns evaluation 
■- * ' ■ 

with 'regulation is likely to prove of greatest significance in extending 

and strengthening the evaluation research domai§. Evidence of 

effectiveness would be required„as a condition for continued 
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public funding of social programs. Some evictenX* of effectiveness 
through program evaluation even might be required pf-j^vately funded 
programs which want tq e^>y tax exempt status . 'The evaluator- 
reeulator Could have authority to suspend public expenditures for — # 
programs and tax exempt status for organizations on the basis of • ■ ^ ^ • 
^ lack of evidence of effectiveness. 

The proposed regulatory model wqjxld extend approaches already 
initiated. In 1965 the Elementary and Secondary, Education Act included 
a requirement ^Eat projects funded through the program be evaluated. 
Federal agencies financing demonstration programs now routinely require 
the -yiclusfbn of an -evaluation component. Some funding agencies go 
a-step further in mandating that evaluates b&conducted by. an agent 
independent of the program operation. ^Tresumably the independent 
evalua/torlnjoys greater leverage in establishing an evaluation' ajfijj£, ^ 
carrying out the evaluation, and publicizing results than ©valuers; .j* . 
subordinate to a program operator* '4^" * , 

' "The mandatory evaluation approach falls short Of $he regulatory 
model proposed here in that it simply requires %t evaluations be. 
conducted. Agencies sponsoring -social programs vary widely in) their , ' 
methodological 'expectations for the evaluations they require. .Further, 

typically there is no requirement that anyone attend seriously to the 

■ n fc 

evaluation results. 

Some precedent -for the regulatory; mod>l also can be' found in 
' accrediting and' licensing strategies' (Glass, 1971). In some sectors 
accreditation is a'cohdition for licensing or receipt of public funds. 
The accreditation approach emphasizes qualifications and facilities. 



Accreditation emphasizes" potential rather than actual performance. 
It is the- shift to an emphasis on methodologically sound evidence of 
achievetmcmt which would differentiate the proposed regulatory * . 
evaluation mbdel from traditional accreditation practice: 

Authority to impose significant sanctions on programs for which 
evidence of effectiveness is lacking' i^ central to the regulatory + 
evaluation model proposed h§re: through legislation, public regulatory 
bo'dies or publicly sanctioned accreditation agencies would be authorized 
to suspend or terminate public funding oft the basis of negative evaluation 
findings. . If established on a federal levef, far example,- a regulatory 

• * r * 

.evaluator would have authority to suspend the payment of federal funds 
t<j, states found to be administering ineffective programs. Further, 
regulatory evaluation bodies would be provided with substantial stfrtlctural 
insulation from routine political pressures. Directors of public 
regulatory evaluation agencies, for example, might be appointed on a 
long term ba^is and bf. subject to removal from office only for gross* 
misM lit. Similarly the legislative' authorization might include 
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'a provision calling for funding based on a fixed formula tied to program 

appropriations in th6 domain to be evaluated. , 

> 

Regulatory ageneies would. set and enforce evaluation research 
standards in their domain. They would articulate minimum sets of outcome 
variables, establish performance standards, define the gfound rules for 
acceptable evaluation methodologies, and supervise the execution 6f 
evaluation studies. * Their ^authority would go beyond* the right to observe 
and exainine records to include conducting experimental studies. 
In conducting randomized- experiments , regulatory evaluators would not 
have ^authority tp deny entitlements. In the case of capped programs, 



- 12 - 



however, they would have authority to employ randomization to withhold 

\ 

services from c.ertain marginal service applicants for experimental 

^valuation purposes, ^ , j 

Some variation in institutional arrangements for conducting 

regulatory evaluation studies, is possible. Regulatory bodies might \ 

authorize evaluations conducted entirely by independent evaluation groups. 

Alternately they might sanction a mixed model in which service 

organizations would maintain internal evaluation units subject to external 

audit. The required audits might be conducted directly by accreditation 

agencies or by authorized independent evaluation groups. As Campbell^ 

(1977) suggests, the external evaluator would review the methodological 

adequacy of measurement procedures, the quality of the data, and the bases 

upon which inferences can be made regarding program effectiveness. In 

most instances the outside evaluator would depend entirely on data . 

collected by inteijia^ evaluators . The outsider might collect some 

additional data* as a check%n or extension of internal data collection 

procedures. Anticipation of a review by external evaluators would 

• create pressure on service organizations to permit sound internal 

evaluation work. Tfie threat of loss of public for 'failure to 

produce sound evaluation data would serve as a strong incentive to service. 

agencies to authorize strong internal evaluation units. 

UJ/timately regulatory evaluators would be expected' to terminate 

programs found to be ineffective. Failure to find evidence of positive 

« 

results on key outcome variables <would be a sufficient basis for 
termination. Regulatory evaluators would not act precipitously in 
closing programs. As a first step, the absence of positive outcome 
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evidence would be'Teviewed with program personnel. » Opportunities 

would bte provided -for refining objectives anfl program strategies. 

\f important new, outcome variables or promising program strategies 

were introduced, new evaluation studies would be conducted. The 

continued absence of positive outcomes, then, would lead to a 

■ 

termination action. 

Making Regulatory Evaluation Work 
fi > 

4rf . No institutional arrangement automatically provides ^fective. 

soluttCJfi^o the problems it was designed to address. Difficulties 

with thg^egulatory evaluation model can be anticipated. The extent 

to wliich they can be overcome will depend on the skill and industry 

of t&ose who work with development of the regulatory evaluation model. 

It is useful ^o consider how some of the inevitable problems might be 

addressed. 

Weaknesses in evaluation methodology will be a source of 
objections to ^regulatory evaluation? Some will argue that while current 
f * evaluation methodologies provide a basis for raising critical questions 

about social progTam effectiveness, they often do not provide the 
conclusive evidence desired as a basis for decisions^ regarding the fate 
* of programs. Some programs, for example, are justified on the basis of 

long term effects which cannot be tested quickly enough to meet the 
requirements of short term decision cy^hs. Other programs are justified 
. on the basis of highly abstract objecti/res which do not readily lend 
themselves to measurement: j RegCiiatory agencies would be expected to 
address these problems on a case by case basis. In some instances 

• . - ■■ • ■ 
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evaluation research requirements might be temjiered^ Jn i^tfcer instances 
reformulation of goal' framewbrks might be require^, Program propondrte*- 
would be required to find a sufficient basis for ^iistifying social 
programs in objectives which are 'relatively inmrediate and sufficiently*- 



concrete so that they lend themselves p'o st^aigh^forwatd measurement % „ 

* / * 

While some might be disappointed ij^the evaluation of programs on the 

basis. of immediate otitcomes, xWuldtoxy evaluators will^be able *to 

' / * / ' * \ 

argue* that immediate outcop6s.4jFe preferable to structUre/aSSk; process 

variables as bases for eva^ation judgments! ^ » \ 

The eagerness of Valuation researchers to conduct true experiinen 

based on randomizatioiy^urely will create concerns among those with 

direct service responsibility. Regulatory bodies will be expected to 

abide by ethical anffNlegal principles protecting the fights of subjects 

of human experimentation. Because of ethical and legal concerns, 

regulatory 'evaluafprs in some instances will have to be satisfied with 

less than opt imaljrese arch designs.' As indicated above, regulatory 

evaluators in thg interest of conducting true experiments could not 

deny individuals? their entitlements. They will have authority to 

insist on an exnferimental design in some form when service resources 

are insufficient to address aggregate needs. Regulatory evaluators 

will have to work with providers to find reasonable ways of reconciling 

service prioritjLes fand experimental evaluation interests* 

* Corrupt icwi * of outcome measures will be a serious continuing 

problem for relulatory evaluators (Campbell, 1977) \ When they know 

evaluation criteria, program operators can be expected to redirect 

their acti^tiis to obtain artificially higji ratings. To the extent 
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that corruption takes the fork of distorted record keeping, evaluators 
can' try to overcome the problem by monitoring record keeping activities 

or by developing independent data systems. When evaluation measures 

i 

cover- only a portion of the goal domain, program operators ean De 
expecteXo concentrate their efforts on the domains which are measured 
at the expense of those which are not. Regulatory evaluators' can contend . 
with this problem by seeking comprehensive outcome measurements or by 
using sampling strategies which are not announced in advance to measure 
portions of a broad set of outcome variables. 

In the case of new program strategies, regulatory evaluators will 
have to be judicious about their timing in introducing outcome evaluations 
Only after program operators have had sufficient .opportunity to solve 
inevitable start-up problems, will it be desirable to introduce the 
.outcome evaluations which will .decide the fate of the innovation. In 
the start-up period, regulatory activities *ill be limited to financial 
audits and analyses of structural arrangements, staffing patterns, and 
service exchanges. In some instances \hese reviews will reveal needs 
for corrective action. In other cases they will provide a basis for 
early termination decisions (Caro, 1977). 

Universal, comprehensive regulatory evaluation will be expensive^ 
In part the high cost of regulatory evaluation will be justified by * 
savings realized through elimination of ineffective programs. In order 
to justify their budgets regulatory agencies will have to demonstrate 
their utility. In part their ability to 'attract sufficient resources 
wil.1 depend on the energy, and'politfkl skills of proponents. At the 
same time the agendas of regulatory evaluators inevitably will exceed 
""available resources. Good judgment will be required in selecting the 
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program issues most deserving intensive evaluation -attention. In some 

instances modest inquiries will be sufficient to surface such ^ ^ 

deficiencies in structure and process that no expensive outcome 

f ; ■ 

evaluation is necessary. Further, evaluators will continue to be 

required to be ingenious in conducting powerful studies with modest 
financial resources. 

. .^-Introduction of regulatory 'evalua£j.on will hasten the 
* v 

professionalization of evaluation research. Standards for evaluation . 
'practice and credentials for evaluators will assume great importance. 
Certification and licensing might be required' for regulatory evaluators. 
In light of the diversity within the field regarding methodological \ 
priorities, formulation of explicit standards for regulatory evaluation 
will provoke great controversies. Premature codification and enforcement , 
of evaluation .standards might commit the evaluation field to practice 
patterns which will constrain its 'long term development. (See, for S 
example, Morell 'and Flaherty, 1978.) In various substantive sectors 
regulatory bodies will be challenged to choose wisely among competing 
methodological claims. They also will be well advised to be alert 
to possibilities for incorporating improvements in evaluation methodology. • 

For some researchers the explicit identification of evaluation. 
<with regulation will be troublesome. Some evaluation researchers prefer 
to see themselves as agents of program development gather than regulation. 
Many evaluators strive to avoid a regulatory identity because the 
obstacles it can create in securing* needed cooperation from program 
personnel. In the long run conscientious regulatory evaluation should 
make service operators a good deal more serious about conducting effective 
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programs.' The greater need to develop programs which will stand up' under 

rigorous testing should stimulate demand for formative evaluation. 

v 

Researchers who, are not willing to work within a regulatory framework 
indirectly, their, may find their opportunities to do formative evaluation 
enhanced. 

Evaluators who work within a regulatory framework will learn to 
contend with overt conflict with program personnel . They will come to 
expect to be greeted with some antagonism and learn to insulate 
themselves from it. Fortunately their" link to progrgn finances will 

put them in a position to \emand information. To gain needed cooperation 

V 

regulatory evaluators will not have to rely fully on the human 
relations techniques employed extensively by evaluators who are 
entirely dependent on ^voluntary cooperation. 

> Regulatory evaluation will expand Substantially the rai^e of 
evaluation findings made available to the public. Particularly for 
programs enjoying inflated reputations, publication of sound evaluation 
findings may lead the public to draw more drastic negative conclusions 
than are justified. A public unprepared for^ews of modest achievements 
may turn against once favored programs. The controversy triggered by 
the Westinghouse Head Start evaluation illustrate^ the problem 
(Evans, 1969). Regulatory bodies w^ll be well advised to prepare the 
public for evaluation reports which may be -particularly troublesome. 
Advance publicity about key evaluation questions and the methods may 
be useful . Responsible discussions of implications of findings should * 
accompany publicity about negative results. 
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In the long run the publication issue may not be serious. Once - 
accustomed to deflated accounts of programs the public may learn to * 
respond without alarm to evaluation reports showing only modest 
progTato accomplishments. Ultimately it should be possible tb develop 
a more sophisticated public which, supports programs whose goals are 

N 

modest but clearly attainable; 

The proposed expansion gjf the methodological scope of regulatory 

agencJ^^iir only add to the importance of issues raised generally 

about regulatory agencies. A number of distortions are possible. 

Regulatory evaluation may come to be dominated by technicians for 

whom methodology becomes its own objective rather than a means of 

t ■ 

guiding p\iblic funding of social programs. Alternately program 
entrepreneurs may find ways of neutralizing the impact of regulatory 
evaluation either by gaining control over regulatory bodies or by 
undermining their political support. If regulatory evaluation is to 
mak e a useful contribution, it must reflect the legitimate concerns 
of the full social program constituency - including taxpayers, program 
operators, practitioners, clients, and evaluation researchers. 
Regulatory'evaluation will do no more than provide evaluation researchers 
with greater leverage in conducting studies and influencing decisions. 
Coiwiderable skill and energy will continue to be required to conduct 
evaluation research effectively even with the advantages of a 
igulatory framework. Evaluators will have to demonstrate their 

/ability to contribute effect iveij^as regulators if the model is to 

\ 

be institutionalized and extendec^. 
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